Variational Inference via Upper Bound Minimization

Variational inference (VI) is widely used as an efficient alternative to Markovchain Monte Carlo. It posits a family of approximating distributions q and findsthe closest member to the exact posterior p. Closeness is usually measured via adivergence D(q||p) from q to p. While successful, this approach also has problems.Notably, it typically leads to underestimation of the posterior variance. In this paperwe propose CHIVI, a black-box variational inference algorithm that minimizesDχ(p||q), the χ-divergence from p to q. CHIVI minimizes an upper bound of themodel evidence, which we term the χ upper bound (CUBO). Minimizing theCUBO leads to improved posterior uncertainty, and it can also be used with theclassical VI lower bound (ELBO) to provide a sandwich estimate of the modelevidence. We study CHIVI on three models: probit regression, Gaussian processclassification, and a Cox process model of basketball plays. When compared toexpectation propagation and classical VI, CHIVI produces better error rates andmore accurate estimates of posterior variance.